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SyS ' em Tha Svmmetnx storage system is configurable to provide data storage for mulUple 

remote copy, can be performed simultaneously ^ 

system s to erform* ~ op^, ^ ^ —ng ^ 

rlirinrs^esno,^ 

SeqU Tl^ an, system « address *e aforementioned problems, as we,, as o»er 
related problems, are therefore desirable. 

jHMiyinPY~ THFIIWENTI0N am pthod and system for automating 

in various embodiments, the invention prov.des a method and system 
In various configured with pattern 

diS c,osed embodiment o, the present nvent^ The ,u^ an 
Mow provide additional example embod,ments and aspects P 
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PIG , is a ,un*na, bloc, diagram o, a computing arrangement configure ,n 
accordance «h one embodiment of the invenSon; 

FIG 2 is a data flow diagram that illustrates, in part, the data exchange 
host, a message processor, and a storage con.ro. system in au.om*g a complex 

multiple host systems in accordance with one embodiment of me invent. 

QFT , IIF nnFScro ^ d examp , e jnvo , ving automatton 

15 BG 1 isafunctionalblocKdiagramofacompuiingarrangementconfigu^din 

- * d " ? ^ti "n 1^ m Depending on the coming recrements of 

Each host controls and provides access ra u« 

,» *** 102 1 provides access to data stored in data storage units 104-1. The 
T I « ™otn are Symmethx storage systems, and vadous administra.on 
25 data storage units 104-1 104 are y . . n8 Forexar nple, storage control 

operations are provided by way of storage contro software 

from EMC. A component of ECC, bKUi-/ 1 imer 

30 and failbacK functions for me Symmetrtx Remote Data FacMy (SRD W 
software, the BCV and SRDF func«ons can be centrally performed for me 
•104-1 - 104-nthat are coupled to the hosts 102-1 - 102-n. 
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TimeFinder software provides a command line interface and also supports sor.pt- 
.asedeZo — . Is.aoo.pie.opera^onc.n.epe.orrnedinatu,,^ 
mod o Zm -tcmated * a scrtpt. However, in ro mpu«ng arrangeme m such as 

«n, administration of me different data storage units, coordinating me d,fferen. 

complex operations further complicates the scenano. 

In order to automate and support concurrent operations**, the 

onger ma expected and Introduce me P oss«y of an error and data conupt^ The 

per system HO is conjured with Single Point Operates (SPO) sof^re from 
Zt The SPO software includes me Single Point Auto-ac*on Message System SP- 

Unisys. 1 1 ice. moccanias to user-defined 
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emulation sess to . Furthermore, .he message ^'TS^eco^ to ^ 

h „s« 102, message processor 152, and the* g ^ 

ana a pattern Tne message proceS sor communicates v*th 

implemented with the SP-AMS soft* ^ ^ 

software on the storage control n „, message 
the host system via a host-specfc interface. At the users 
processor initiates a dialogue with the storage control system. 

Using database maintenance software that accompanies the fv*' ' 

operation commences with the message processor send.ng a comma 

tnatthemessagepr.^ 

system. A command is dequeued and sent when the term, 

command prompt message from the storage control mistral *e 



storag e u*. ft. storage control system 108 is coupled to ft. service processor via a direct 

cable connection. 

Depending on the particular type of host system 102, the pattern database : may 
include additional patterns and associated responses for other messages sent to the host, 
kelpie, during fte course o, performing ft. comp,ex opera U on, - 
may generate various status messages indicating fte results of commands ,ssued .0 fte 

storage control system. *^ M f rtr 
FIG 3 is a flowchart of a process for automata concur, complex operates or 
multiple host systems in accordance wift on. embodiment of .he invent. The genera 
control flow wil, be described «ft ^ference ,0 fte Appendix, which more spec^ca* ts 
forft the patterns and associated response definitions that implement automaton of fte 

complex^ope^ratioa^ ^ p a ^ ern database is created and activated. As previously discussed, 
in the example embodiment the pattern database , created using .he SP-AMS software. 

1 database sped.ica.ion se, forth in fte Appendix implements fte remote cop, ope* n 

2 Symmeftx dis k storage systems ft* are coupled .0 a Unisys IX data process,ng system, 

or multiple partitions thereof. 

A. step 154, a terminal emulation session is established between fte message 
processor and fte storage control system. This allows the message processor ,0 submit 
commands via a command line interface and receive messages from 
system. The message processor a,so establishes communions w,ft fte host s, tep 154. 
The particular interface through which communions are established wrth fte hos ,s 
system dependent For example, fte interface .0 a Unisys IX system is via a console 

'"he example database, a complex operation begins when fte hos. sends an ^ 
initiation message .0 .he message processor. The ini^on message - * 
either a. a scheduled (programmed, time or manually via operator ,n P ut (ad ho* When 
initiation message is matched, the associated response definite is processed, and 
Tld is either immediately submitted ,0 fte storage control system , there are no 
, Imands waiting to be issued or queued i, ftere are commands waittng .0 be 

,n one embodiment, four command queues are deflned and used ,n fte response 
deflations for selected patterns in fte pattern database. The queues indude, in order o, 
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high es, ,o lowest precedence, me flash queue, me Immediate queue, .he priority queue^and 
the routine queue. The flash queue Is reserved for the INSTANT SPLIT command (operatm 
on the local BCV group), from which it is executed instantly. The immediate queue ,s mean, 
for requests to start another remote copy process. 

The command queues are implemented as character string variables as illustrated a, 
lines 50-77 of me Appendix. The commands w,.hin each command queue are delimited by 
me ■ - character, as illustrated at line 370 of me Appendix. 

~ At step 158, the message processor processes command prompt messages from the 
storage control system. The response associated w«h me command prompt pattern .n ^.he 
pattern database species removing a command from a command queue and subm,tt,ng me 
command to me storage control system. 

The queues are processed in precedence order. That is, me flash queue must be 
empty before commands in the immediate queue are proceed, the immediate queue mus 
be empty before commands in me priority queue are processed, and the priority queue must 
be empty before commands in me routine queue are processed. WWn each queue, the 
commands are processed in first-in-first-out order. 

For other messages from the storage control system, at step 160 the process 
matches me messages against patterns in me database and queues the associated 
command responses to a command queue. 

At step 156, the process steps 156, 158, and 160 are repeated unt,l me entore 

process is complete. 

The descriptor below further explains me automated remote copy operation as 

exemplified in me database specification in the Appendix. 

The Remote Copy Process (RC) is a means by whi* data is moved 
and error-free from one set of EMC disks to another. The scripts utilize SP-AMS to match 
me ssages sen. to a telnet session, which is a DOS window into the Symmetrix NT system. 

SP-AMS software provides control for SYMCLI commands that are sent by me 
automation. In mis app,ica.ion. a console message from a host provides me .rigger to star, 
the Remote Copy. The message is sent to me console in the following form: 
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START DATA TRANSFER DISKGROUP <bcv_groupid: 
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The BCV groupid refers to a grouping of disks: Disks with the same groupid are 
oper a tedonatthe S ametime;the S ubjectofBCVsp.itandestab.ish,etc. The algorithm for 

complete execution of the RC process follows. The SP-AMS and AMS patterns respons.ble 
for the automation are listed with each step. Note that the example is applicable to a Un.sys 

5 IX system, where operator console automation is provided by way of an AMS database, 
which is separate from the SP-AMS database and is hosted on an operations console of the 
Unisys IX system. "AMS" refers to the database and/or software on the Unisys IX console, 
and SP-AMS refers to the database and/or software on the operations system 110. In «,s 
example the pattern database 154 of the operations system operates in conjunction w,th a 

10 second pattern database configured for the operations console of the host. 

1 Host message starts Remote Copy; console sends message to SPO - start the 
process for the named <bcv_groupid>. 

15 AMS: Start.RC 1 

SP-AMS: RC_Start 1 

2. Verify all remote BCV split; verify local synchronization. 

20 SP-AMS: RemNotSplit 1 

RemNotSplit2 
RemSplitConf 1 



25 



30 



35 



40 



3. Verify all local BCV established. 

SP-AMS: LocNotEst 1 
LocNotEst 2 
LocEstConf 1 

4. Loop until command lock free and no RDF in progress. 
SP-AMS: LocEstConf 1 

5. Lock for Pauseio. 
SP-AMS: LocEstConf 1 

6. Issue Pauseio command to host console. 

SP-AMS: LocEstConf 1 
AMS: PauselO_Done 1 



8 



Patent Application 

Attorney's Docket No. RA-5388 F„prsss Mail FFM7780257US 

05/15/2001 

7. Host console sends instant split command to SPO for forwarding on to symm.host. 

AMS' PauselO_Done 1 

PauselO_Done 2 
5 PauselO_Done 3 

SP-AMS: LocSplitSent 1 

10 period). 

SP-AMS: ResumelOl 
AMS- AMS_Answer 1 

ResumelODone 1 

15 ■ , i, 

9. Free Pauseio lock. 

SP-AMS: Resumel0 1 
20 10. Verify all local split. 

SP-AMS: Resumel0 1 
LocSplitSent 2 
LocSplitSent 3 
25 LocSplitConf 1 

11 Initiate RDF establish. A good copy of <bcv_groupid> has been received at the 
' remote site (R1-R2). Set RDF in progress lock. 



30 



40 



SP-AMS: LocSplitConf 1 
12. RDF all synchronized. Initiate RDF split. Free RDF in progress lock. 



SP-AMS: RDFJnProg 1 
RDFJnProg 2 
RDFJnProg 3 
RDFJnProg 4 
RDF_Est 1 



13. RDF all split; initiate local BCV establish. 



SP-AMS- RDFSplitSent 1 
RDFSplitSent 2 
RDFSplitSent 3 
45 RDFSplitConf 1 

14. Local BCV all synchronized; initiate remote BCV establish. 
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SP-AMS: LocEstSent 1 
LocEstSent 2 
LocEstSent 3 
Local EstConf 1 



15. Remote BCV all synchronized; initiate remote BCV split. 



SP-AMS: Rem_BCV_Est 1 
in Rem_BCV_Est2 

Rem_BCV_Est3 
Rem_BCV_Est4 



16. Remote BCV all split. Remote Copy complete - notify console. Clean up. 



SP-AMS: RemSplitSent 1 
RemSplitSent 2 
RemSplitSent 3 
RemSplitDone 1 
20 RC_Comp 1 

Cleanup 1 
CheckVars 1 



The Remote Copy process has been designed to allow multiple partitions to move 
data simultaneous,,. While the symmhos. te.ne. session can only process one command a. 
a time the Symmetnx Timefinder/SRDF faciMty allows more than one BCV/RDF 9 roup o be 
addressed and to be split or restored in parallel. This requires a queueing mechan,sm to 
store the commands and to retrave and execute them when the command prompt . 

returned to the telnet session. 

A global ■Command.Lock- (variable) is given to the first to come to be served 
(FGFS) Commands mat arrive While another command is in exe^on are queued to one of 
our prioritized command queues, Flash (highest precedence). Immediate, Priority or Routine 
(lowest precedence). The Flash queue is reserved for the INSTANT SPLIT command 
(operating on the local BCV group), from which it is executed instantly. The Immediate 
qu eue is meant for requests to star, anotoer RC process. The Priority queue holds most of 
the commands. Among the queues, each command is processed in the same order . was 
added to the queue; all commands in higher queues are executed before moving to the 

lower-precedence queue. 

When the host console message triggers SP-AMS to start a remote copy, the remote 

copy begins what will include a 6-second race to the "split": 

10 



Host console sends message to SPO - okay to start Remote Copy 
At an optimum time, SP-AMS sends "pauseio" command to IX host. 
OS2200 "Pause Disk I/O for ClearPath OS2200" feature 



o 

invoked 

o OS2200 stops all writes to disk 



R/R console message indicates disk paused time; must not exceed 20 
seconds from now until an automated answer is received 



Then 



the "critical period" begins (expect between 6- and 12- second delay). 

In a flash, the SYMCLI command is executed to cause the instant split 

of an EMC Symmetrix device group 

o In the initial device split to start the process, the BCV - R1 is 

created. 

o EMC feature instant split confirms all disk writes immediately 
allowing the host to resume normal operation; disk writes are trickled in. 
o SPO detects confirmation of instant split command, sends 

DONE to console. 

host console AMS patterns provide a mechanism to answer the R/R 

console message. 

This ends the "critical period." 

In the event of a problem during the instant split (i.e. after the pauseio command is 
issued) it would be possible to have the operating system and it's disk I/O paused forever. 
This potential requires a timing mechanism be in place to automatically answer the read-and- 
reply message after a certain amount of time. Should this timer expire and the mechan.sm 
trigger the answer, the Remote Copy process is assumed to have failed. 

This mechanism is a timed ANSWER action in the AMS database active on the host 
console. The pattern, PauselO_Done 1, is listed below. Note the use of the WAIT 20 
SECONDS clause with the ANSWER action: 

D ^S U ^ 0 resTme 1 all d,sk „o when done, enter done, else abort:- 

/* -i _*„2— *-3-*-4~*-5-*-6~*~7~*--8«*--9~M 0-*- 11--/ 

11 



Attorney's Docket No. RA-5388 
05/15/2001 



Patent Application 
Fvpress Mail FP387780257US 



TYPE ANY-SENDER READ-AND-REPLY 
INSTANCE PRIMARY 
PRIORITY 128 

TOKEN KEYWORD 2 "RESUME 
5 TOKEN FIXED 4 "DISK- 
TOKEN FIXED 7 "DONE," 
TOKEN FIXED 11 "ABORT:" 
ACTION ALL HIGHLIGHT "WHITE" 

10 ^|^S£he VTO RESUME ALL DISK ,/OV message!" 

A °command=!r^ 

instant split" 
ENDIF 

S^ |n 7^ d .:.^ R T; r nMn S AMSWFR ^..serndV-WRT" 

20 IF Console = "RESPONSE" 

0_Done_1 |TEXT=\_MESSAGE\" 
ENDIF 
25 END 

When the RC process operates normally, the message is answered by the 
automation within the 20 seconds. The deadman's switch timer always expires. When the 
mechanism attempts to answer the message that is no .onger outstanding at the console, no 
30 runtime error is incurred and the attempted action is ignored. 

The Remote Copy process has a timerthat exists on the host console. When an RC 
process is started, the variable "RC.Timef is set to some non-null value. When the RC 
process completes, the variable is reset to nul.. The variab.e definition (shown below) has a 
TIMEOUT value of 60 minutes: 

TYPE STRING RETAINED 
TIMEOUT 60 MINUTES 
DEFAULT "" 
40 END 

Another variable, RC In.Progress, is set to the BCV groupid that is in progress. 
When the RC Timer and RCJn.Progress variables are first set to indicate the start of the 

12 
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process, *e message pafern Start.RC 1 sends ft. "OM.M CHECK RC TIMER" command 
with a one minute delay: 
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*/ 



r This message will normally come from a SAM (SMA) job 
* o n ?TART DATA TRANSFER DISKGROUP TAC */ 

/* i *-2— *-3~* — 4 — * 5 */ 

TYPE PRIVILEGED-EXEC OTHER 
INSTANCE ANY-INSTANCE 
PRIORITY 128 

TOKEN KEYWORD 2 "REMOTE- 
TOKEN FIXED 4 "DISKGROUP" 
TOKEN MASKED 1 "\*8\START" 
RESET RCLAST 
SET color = "WHITE" 
IF RC_ln_Progress != "" 
IF Console = "RESPONSE" 

MID=RC Already_ln_Progress|" \ 

No action taken." 
ENDIF 

!It1wTll^ISRA^ "Remote copy already in progress. No action taken." 
ELSE 

RESET SPO_Resumed 

SET bcv_groupid = "\_MESSAGE { } -1\ 

SET RC_ln_Progress = "TRUE" 

IF Console = "RESPONSE" 

■typeIcc^cK 

$HOST$" 

I* Now kick off the console timer check for the process: */ 
SET RC_Timer = "TRUE" 

' F^ n y r i; R SSS N ^, M .m=« TCOMMMn "OMMCHF CKRC TIMER" 

ENDIF 
ENDIF 

ACTION ALL HIGHLIGHT "\color\" 
IF Console = "RESPONSE" 
ACTION ALL EVENT-REPORT \ 
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.. TYPE =LG|C L ASS=Ho S ,|,NSTANC^Lo g Na m eMAPPL= $ CONSOUE $ |APPLQUAL=S Brt . R 

C_1\_RUNID\|TEXT=\_MESSAGE\" 

ENDIF 
5 END 

, f the variable is reset in sixty minutes, the recurring timing message issued 
If the vanaDie u ' hQ RP , n p r0 aress and finds a non-null value, the 

detectsthis. lniW.cM.^«ch^th.RCJ^PW«« 
RC process is declared error-terminated and an alarm ,s sent to SPO (see RC_ 

10 below). 

DEFINE "RC Timer" 1 
MESSAGE T8\CHECK RC TIMER" 
I* ...-1— -*2-*-3-*/ 
TYPE PRIVILEGED-EXEC OTHER 
15 INSTANCE ANY-INSTANCE 
PRIORITY 128 
TOKEN KEYWORD 2 "RC 
TOKEN FIXED 3 "TIMER" 
TOKEN MASKED 1 T8\CHECK" 
20 SET color = "WHITE" 

ACTION ALL SUPPRESS 
SET RC_Timer_Loop = RC_Timer_Loop + 1 
IF RC_Timer_Loop = 32767 
RESET RC_Timer_Loop 
25 ENDIF 

IF RC Timer ='": e stj „ jn progr ess: */ 

/* that means we timed out-checK to see ■■ w<= 

IF '" p ™n ress l= "" qpo d stop th e automation! */ 

r then we are still in progress-alarm SPO ana stop me 

30 IF Console = "RESPONSE" 

weSoT^ UP 

FAIL" 

Ar-nnM ALL EV »=MT-REPORT\ 

,, rr tlin?T1 ;|A r p| ^ nRMAME\ISEV =maiorlALA 

40 "«»«♦»■ \RCLASIV 

ENDIF 

S A^™V "Re™, copy S d ou. -or disKgroup ^ 
ACTION ALL DISPLAY "Last state: \RCLAST\ 

45 'S^S •° m ' m CLEAN UP ^ 
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ENDIF 
ENDIF 
ELSE 

IF RC_ln_Progress != "" 
'aCtTo^aIl^I 8 MINUTES TCOMMAND "OM,M CHECK RC TIMER" 
ENDIF 

IF RC_Timer_Loop % 6 = 0 

ACTI0^ALL^ISP E LAY "Remote copy in progress for disk group \bcv_groupid\ ..." 
ENDIF 

r d^nothing to recheck the timer since we are done with the remote copy. */ 
IF Console = "RESPONSE" 

ACTION ALL TCOMMAND "OM,M CLEAN UP \bcv_groupid\ 
ENDIF 
ENDIF 
ENDIF 

ACTION ALL HIGHLIGHT "\color\" 
IF Console = "RESPONSE" 

■T?PE™G|C^ 
er_1 |TEXT=\_MESSAGE\" 
ENDIF 
END 

Accordingly, the present invention provides, among other aspects, a system and 
method for automating complex operations of a computing arrangement. Other aspects and 
embodiments of the present invention will be apparent to those skilled in the art from 
consideration of the specification and practice of the invention disclosed herein. It is 
intended that the specification and illustrated embodiments be considered as examples only, 
with a true scope and spirit of the invention being indicated by the following claims. 
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